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inserted together with the watermark information. When 
the moving image has been brought to a standstill into a 
static image, the watermark information can be visually 
perceived with ease, so that illegality such as piracy can 
be readily exposed. Thus, the present invention can 
provide a simple and inexpensive scheme for the copy- 
right protection of the moving image. 
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Description 

roooil The present invention relates to the processing of moving images. More particularly, it relates to technologies 
o^nsir^^^^^ 

S ~te JKS^fS2T?^ to techno.ogies for copyright administration services for the moving .mages 
2JS trl^M^XheaZ^o distribution methods, and technologies for watermark information authentication 
2^yT5^rJ5?iS^. it relates to an insertion system, an insertion method and a storage medium 

Tc^r™ hod can 

E55 ^eScTbSound which has given birth to the present invention is a situation where anybody can 

12 S a work Dv utilizing a personal computer, owing to the spread of digitization, the popularization of the 

t^^^S^S^C^. MOs, DVDs etc.. and where problems concerning copyrights have become 

(a) Analog moving image packages: Cinema, video tape, etc. 

S3 SS^SoS J2.2££t cabf e TV. sate,,e broadcast, etc. 
S Delving image broadcasts: Digital TV. digital cable TV. digital satellite broadcast, etc. 
(e) Digital moving image communications: Internet communications, etc. 

100061 All of these technical fields handle moving images as their contents. Recently, the analog form and the digital 

rejected in the first place, and how piracy is prevented. 
[0007] Mentioned as prior-art techniques relevant to these subjects are the followmg. 

(1) Scheme wherein, using copy restriction codes, the video recording duplication and reproduction display of mov- 

Sagl date 'iSSrJd a control storage area for storing control information therein, the latter of which ,s hardware 
nrotected thereby to prevent the moving image data from being altered by general users. ^ nro ^ 

,52 1 as MPEG) wherein a digital moving image is subjected to data compress,on so as to store or 

45 S^nsm^ng a moving image, it is encrypted in order that only specific allowed recipients 

* be — * u ™ b,e (an in ^ we watermark) is 

Ser^ into me data of a moving image so as to transmit or circulate the resulted movmg ,mage. 

[00081 Even with these schemes, however, a satisfactory scheme for the copyright protects ° n J* s ™^^ 
ZLeJvet This is also obvious from "Call for Proposals" issued under the date of Ju y 1 . 1997 by he Date Hdmg 
ffS nufmHSG of the Copy Protection Technical Working Group (CPTWG). The call for proposals invitee public 
SubGroup <°"^° , | ™^ d ^SS e scnemes for the copyright protection which employ various "watermark" tech- 

55 nTesTn" 

lo^Note 0 •■Ca5 foTp'roposals", Data Hiding SubGroup, Copy Protection Technical Working Group. Version 1 .0. July 
1 , 1997 at http://www.dvcc.com/dhsg 
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[0010] The CPTWG is an association of traders concerned with digital video recording and reproduction, and tech- 
nologies to be standardized in the trade are taken up here. Such a technology represents a present-day technological 
level in the field concerned. The proposals to the DHSG which is one of the subgroups of the CPTWG will contribute to 
the technical standard of the field concerned. Concretely, after being studied as a more comprehensive technology by 
5 the DTDG which is another subgroup, the proposals are expected to be reflected in the standard of a secure bus termed 
"IEEE1394". 

[001 1 ] The present invention complies also with the Call for Proposals. In this regard, the prior art of electronic water- 
marking will be stated below. 

[001 2] There have hitherto been practiced copyright administration services for still images and music that utilize elec- 
w tronic watermarks. The electronic watermarking technique in the prior art is such that the ID of a copyright holder, the 
ID of image data, the ID of the recipient of the image data, the ID of a dealer mediating circulation, the ID of a user, etc. 
are embedded in accordance with the kind of the digital work. As an expedient for the technique, the image is circulated 
through the path of the Internet, and the ID numbers are open to anybody. By way of example, products "PictureMarc" 
(Digimarc Corporation in the U. S.) and "FBI Pro" (High Water Signum Ltd. in the U. K.) correspond to the expedient. As 
is another expedient, the image is circulated by utilizing the Internet or the like, and the ID numbers are managed by key 
data. By way of example, products "SysCop" (Fraunhofer Center for Research Computer Graphics in the U. S.), "Tiger 
Marc Image/Data Blade" (NEC America in the U. S.) and "Argent" (Dice Co. in the U. S.) correspond to this other expe- 
dient. 

[0013] Regarding researches on copyright administration schemes for moving images that utilize electronic water- 

C2o marks, various methods in which a bandwidth compression technique (MPEG2) for the moving image is combined with 
) the electronic watermarking have been proposed, and they are basically classified into two types. The first of these 
types is a data hiding scheme wherein electronic watermark information is inserted in a spatial frequency region so as 
to spread the watermark information in a broad spectrum region. The second type is a scheme wherein the electronic 
watermark is superposed on and inserted into sampled value regions of two dimensions or three dimensions. With the 
25 first type, the watermark information is difficult to remove, so that the intensity of the watermarking is high, but the image 
quality or tone quality of the original data degrades conspicuously. Moreover, frequencies need to be converted twice 
(as conversion and inversion). Therefore, the burden of processing on the first type is heavier than that on the second 
type wherein the watermark is embedded in the sampled value regions. Accordingly, the first type requires dedicated 
hardware. The basic technical themes of the electronic watermarking are (1) that the watermark information is embed- 
30 ded with the degradation to the image quality or the tone quality avoided to the utmost, and (2) that the embedded 
watermark information is made difficult to remove to the utmost. 

[0014] In the scheme wherein the watermark information is embedded in the frequency components, it is embedded 
in the high frequency components of those parts of image data or voice data whose changes are inconspicuous to the 
human eye, for example, the parts of the hair of the head and the contour of an object, in order to suppress the degra- 
35 dation. When the quantity of the watermark information data to be embedded is increased and is spread in the principal 
components of the original data, the intensity of the electronic watermarking becomes high. However, when the spread 
of the watermark information is too broad, the image quality or the tone quality degrades. Moreover, the watermark 
sometimes disappears or deteriorates due to MPEG encoding/decoding. 

[001 5] Also in the scheme wherein the watermark information is embedded in the sampled value regions, it is embed- 
Qyo ded in redundant parts so as not to be seen or heard as noise. By utilizing the visual characteristics and aural charac- 
teristics of man, the quantity of watermark data to be embedded in the redundant parts is made large, and the quantity 
of the watermark information in the principal parts of the content is made small. Thus, the watermark information is 
embedded inconspicuously. 

[0016] There are also different schemes wherein watermark information is embedded by utilizing the existing tech- 
45 niques for image bandwidth compression. By way of example, Fraunhofer CRCG (in U. S.) has developed the scheme 
wherein, in embedding the watermark information in the process of the compression of the JPEG, that is, at the stage 
of linear quantization, a medium frequency coefficient among DCT output coefficients is varied, and the variation is 
used as the watermark information. Besides, NTT (in Japan) or NEC (in Japan) has developed the scheme wherein, 
when the conversion coefficient of DCT is to be linearly quantized, a coefficient value is changed little by little so as to 
so form the watermark information. In addition, Mitsubishi Denki (in Japan) is developing conjointly with Kyushu University 
(in Japan), electronic watermarking which employs wavelet conversion. Kyoto Institute of Technology (in Japan) has 
proposed the scheme wherein the watermark information is embedded in the moving vectors of MPEG. Concretely, a 
table in which the watermark bits of 100 bits are held in correspondence with the individual moving vectors existent in 
the number of 330 per frame, is prepared as secret information at each of transmission and reception ends. It is secretly 
55 utilized as a code table to check whether the watermark information is true or false. 

[0017] Straightforwardly, problems related to the present invention are stated in the Call for Proposals made by the 
CPTWG/DHSG. In the Call for Proposals, a recording system in the package medium of a moving image and a trans- 
mission system in broadcast/communication which satisfy the following specifications are mentioned as below: 
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Permittina a content offerer to put a "DHS watermark" indicative of copyright into a digital video medium for 
ZS^Ti^i the copyright and preventing illegal video recording and reproduction. Here, j « 
esp^ a burner from easily copying the moving image the duphcafon of wh,ch is 

prohibited. 

r0018l A digital video equipment in the future will incorporate this standard and will function so as to "-eject illicit video 
SloXSn W this standard is chiefly directed toward DVDs, it .s similarly applicable to other 

blowing pointe are requested as properties/capabilities which the new systems are to possess: 

(1) When the moving image is reproduced and displayed, a viewer does not have an offensive feeling. 
5>J The coov restriction mode is easily detected by a video recording/reproduction device. 

U ™ V5$L SSon of the DHS watermark is easy in various areas (moving image data, compressed data. 
^ZT^TZ^n detect the one-copy mode, and can set a new disk tor recording to the no- 
more-copy mode. 

(5) The erroneous rejection of recording is reduced. 

6) The detection rate of the "DHS watermark" is high (for example, at least 50 A). 

(7) The DHS watermark is held in ordinary video processing which is executed by a viewer. 

f«\ a license is allowed under reasonable conditions. 

2 t\S : new systemslan be exported to and imported from Japan. U. S. A Europe, and other countr.es. 
llVi The new systems are technologically mature, and can be demonstrated. anxMma 
1?) Swing to tS DHS watermark, the three kinds of copy control modes can be detected continuously anytime. 

The DHS watermark can be inserted efficiently by real-time processing. 
(13) to encode^eSer for the DHS watermark is capable of processing at a DVD operafon speed. 

[0020] The present invention is intended to incarnate a better scheme for copyright protection by addressing the 
above aims and the specifications relevant to the three kinds of copy control 

mo2il The oroblem of the prior art concerning electronic-watermark insertion techniques is as explained below, tven 
nrilTart schemes His aimed at to embed the watermark information with the degradation of the image quality or 
SltSi^SSS to thTu^moil I and to make the embedded watermark information difficult of remove o the 
S^T^.™ however, incur the problem that considerably complicated processing ,s necessitated, but that a 

1!d02T7 wat e erma?k insertion device according to the present invention is premised on the insertion of watermark 

SSSr ,D ?S^^ selects frames into which the watermark information is to be inserted, at irregular 

51 ,T om among ^the p'u^y of frames. The selection of the frames is effected in such a way that the frames are 
dlteTm ined by the use of for example, random number values which are generated under a control^ condrtion. 

SSS? 'Besides the watermark insertion device according to the present invention can be so constructed as to further 
[0027] Besides, tne wnnran * watermark information has 

Snsert^^^^^ 
SdTcavl^ 

mage To Tage uTw be so constructed as to further store therein information indicative of those Pfi***" * e 
Zes S Z> watermark has been inserted, or a condition for generating random number values wh.ch the water- 
mark-information insertion unit uses for determining the insertion positions. 
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[0028] Also, the watermark insertion device according to the present invention can be so constructed as to further 

comprise a copy-restriction-information insertion unit by which information indicative of a restriction on the copying of 

the moving image is inserted into an area where control data for the moving image are recorded. 

[0029] Moreover, the present invention shall cover within its scope a method of generating the moving image in which 

the watermark is inserted, a storage medium which stores therein the moving image or a program for implementing the 

functions of the above device by means of a computer, and an apparatus which reproduces the contents of the storage 

medium. 

[0030] According to the present invention, a simple and inexpensive scheme for the copyright protection of the moving 
image can be provided. 

[0031 ] Specific embodiments of the invention will now be described, purely by way of example, with reference to the 
accompanying drawings in which: 

Fig. 1 is a table showing the specif icaations of copy control modes which conform to the CPTWG/DHSG; 

Fig. 2 is a block diagram showing the fundamental construction of the present invention; 

Fig. 3 is a block diagram showing the second fundamental construction of the present invention; 

Fig. 4 is a diagram for explaining the basic operations of a device for inserting a visible watermark intended for a 

moving image; 

Fig. 5 is a flowchart showing the basic flow of a watermark insertion process; 

Fig. 6 is a diagram for explaining a method of determining frame numbers; 

Fig. 7 is a diagram exemplifying the determined results of copyright information insertion frames: 

Fig. 8 is a diagram showing the determined results of copyright information insertion frames in the case where the 

rate of insertion has been changed; 

Fig. 9 is a diagram for explaining a method of determining the insertion positions of copyright information; 

Fig. 10 is a diagram showing a practicable example of the copyright information; 

Fig. 1 1 is a diagram for explaining the insertion of the copyright information into one frame; 

Figs. 12 and 13 are flowcharts showing the detailed flow of the visible-watermark insertion process; 

Fig. 14 is a diagram showing a moving image file in which control information is encrypted and borne; 

Fig. 15 is a flowchart showing a process for restoring a watermarked moving image into an original image which is 

not watermarked; 

Fig. 16 is a diagram for explaining a copyright indication method which serves to exclude the influence of a sublim- 
inal effect; 

Fig. 17 is a diagram showing an example in which a different pattern is inserted as part of copyright indication; 
Fig. 18 is a diagram showing an example in which a pattern obtained by encrypting frame number is inserted as 
part of copyright indication; 

Fig. 19 is a diagram for explaining camouflagic examples in the case of the piracy of a moving image; 

Fig. 20 is a diagram for explaining a method of disputing with an unlawful dealer who has pirated a moving image, 

about the legality of copyright; 

Figs. 21 and 22 are flowcharts showing a process for verifying copyright against a moving image of suspected pla- 
giary; 

Fig. 23 is a diagram for explaining examples of synthetic decisions on the plagiary of the moving image; 
Fig. 24 is a table showing four kinds of copy control modes; 

Fig. 25 is a diagram for explaining the basic operations of a system for inserting watermark information and copy 
restriction information; 

Fig. 26 is a diagram for explaining the operation of a moving-image recording/reproduction apparatus as complies 
with a request for recording or reproducing the contents of a storage medium; 

Fig. 27 is a diagram for explaining a copyright protection scheme which employs both copy restriction information 
and watermark information; 

Fig. 28 is a table exemplifying copy control codes which serve to realize the copy control modes; 
Fig. 29 is a block diagram showing the structure of a readout/reception device; 
Fig. 30 is a diagram showing the video format of the MPEG; 

Fig. 31 is a flowchart showing a process which is executed using extended copy control modes; 

Fig. 32 is a diagram for explaining the course of the creation, circulation, and video reproduction/recording of a 

package medium; 

Fig. 33 is a block diagram showing the architecture of a computer system which incarnates a watermark insertion 
device; and 

Fig. 34 is a diagram for explaining the loading of a watermark inserting program into a computer. 
[0032] Fig. 2 is a block diagram showing the first fundamental construction of the present invention. In more detail, 
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™er^^,r^ ln me 

[0034] A waternwMnlormetion .nserton una * '"serBWe "J""** ™abo whlc ° ha „ e been selected by Ibe 
StSC ps^=^^<""*" ' ' ^S, pe^Sbt- hbege. However. ft. 

description, such a watermark shall be called the "visible watermark 

[0036] Even when the frames into which the visible ^T^^^^^ me poSion of each frame 
to 100. the number of each frame " ha **^ information in each 

among the 100 frames « determmed at rand ° m - Besi ^' *,I' r i s tended in detecting all of the frames which have 
frame is determined at random. In ^^^^^ SSta ta been inserted at the random posi- 
been selected at the irregular intervals and in which the ^termartt info ™™ n a when the ; of 

tions. from among the large number of frames which const of the watermark 
the moving image is intended, a considerably ^'^ c Ss in Jl Smes bearing the information, 

information, (for example copyright .nformatoon) as based I on searches m all * e ^ 9 encrypti ng the numbers 

25 [0037] In one aspect of performance of the presen ^J^'^S^Sto In ttS frames at which L watermark 
of the frames in which the ■"^'£7?^ the watermark information is 

information is inserted, can be stored and ^ ^? ned J encrypting the controlled condition for 

inserted. Alternatively, it is possible to store and keep •nfor'nafcor ^^^^J -, s t0 be inserted , and/0 r the 
random.y selecting the frame numbers of the frames <nto wh^h * e "^^^ information is to be 

confr^d™ 

the legal settlement of the copyright problem watermark information in the frames 

[0038] In another aspect of performance, partial image ' Us aSoTngly permitted to restore the 

because of the insertion of the watermark information can 3h *• a *> ve stored results 

S^n^^^^ 

sr.™ 

run a computer program. According to the present '"^^ is sto red in any of various 

rr^eT^^^^ 

present invention. examDle a DVD drive) is enabled to permit or 

[0040] Further, according to the present invention, a med,um n d ^ h ( ~ r ^ a 7 h P 'f / " restri ition code is previously 
bar the video recording or duplication -J^XCirorlnt^ data. More- 
recorded in a control-data storage area ^ * e .^3j™^2S5S. copyright of the moving image work is 
over, in the data area of the moving .mage, the information " l ™ "gL 8 ^ copyright indication may not 

embedded in the large number of frames c0 " s ^^ it may be visually 

visually recognized in an ordinary mov.ng-.mage d to play °"be exposed or can be prevented from 

perceivable clearly in a frame feed display. ™- "^^^^^SljS^ (dupScation. A-D©- 

Sr^nSerr^ 
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of the creation of one or more copies, the allowable number of copies in the case of the permission, etc.. in an area in 
which various control data are recorded for a plurality of frames constituting the moving image, for example, which is 
protected from an alteration by a user, unlike an area in which moving image data themselves are recorded. 
[0042J The fundamental construction shown in Fig. 3 operates in a sequence in which the copy restriction information 
is inserted after watermark information, for example, copyright indication has been inserted by a watermark-information 
insertion unit 2 into the frame which had been selected by a frame selection unit 1 . However, the aspect of performance 
of the present invention is not restricted to the exemplified sequence, but it is also possible to insert the watermark infor- 
mation after the insertion of the copy restriction information or to simultaneously insert the copy restriction information 
and the watermark information. 

[0043] The moving image in which the watermark information and the copy restriction information have been inserted 
in accordance with the present invention, is stored in any of various storage media and is distributed in the market, like- 
wise to the moving image in which only the visible watermark has been inserted. 

[0044] The moving image created in the present invention is delivered to the hands of a consumer by way of example 
in the form of, for example, any storage medium, and it is reproduced or recorded (for duplication). On this occasion 
according to the present invention, the execution of the video recording is controlled hardware-wise on the basis of the 
contents of the copy restriction information. 
[0045] More specifically, according to the present invention, a video recording/reproducing apparatus for the moving 
image decides the contents of the copy restriction information hardware-wise or software-wise, and it determines 
whether the video recording is rejected or executed, in accordance with the decided results. The contents of the copy 
restriction information include the permission or rejection of the duplication and the number of duplicatable copies (chil- 
dren) as mentioned before. As a further content, it is possible to designate, for example, a condition for re-duplicating 
or recopying the duplicated moving image, in other words, the number of generations. 

[0046] As thus far described, according to the present invention, the visible watermark and the copy restriction infor- 
mation are inserted into the moving image composed of the plurality of frames, thereby to facilitate the prevention and 
25 detection of any illicit copy. 

[0047] There will now be described an aspect of performance for inserting watermark information in the present inven- 
tion in correspondence with the fundamental construction shown in Fig. 2. 

[0048] Fig. 4 is a diagram for explaining the basic operations of a system for inserting the visible watermark intended 
for a moving image, in this aspect of performance. Referring to the figure, the watermark insertion device 10 of the 
present invention creates a watermarked moving image 1 3 in such a way that watermark information 1 2 is inserted into 
the frames of the original image 1 1 of the moving image as have been selected at irregular intervals, in the form in which 
the watermark information 1 2 is visually perceivable clearly in a still image, but it is not visually recognizable in the mov- 
ing image. The watermarked moving image 1 3 is sent to the side of a user via any of various circulation processes and 
it is reproduced using, for example, a video playback apparatus on the user side. The watermark cannot be recognized 
in an ordinary reproduction mode, whereas it can be recognized in frame feed reproduction or in slow reproduction 
[0049] In the present invention, naturally any watermark information can be employed as the visible watermark which 
is to be inserted into the original moving image. In the ensuing description, a case where the indication of copyright for 
the moving image is inserted as the watermark information shall be mentioned as a practicable example It is permitted 
^ to expose illegality such as the piracy of the moving image and to protect the copyright, by inserting the copyright indi- 
go cation into the moving image. An apparatus therefor needs to fulfill the following conditions (a) ~ (h): 

(a) The information of the copyright can be readily embedded in the moving image. 
A load on a process for the embedment is not heavy. 

(b) In the ordinary looking and listening of the moving image, the quality of the moving image is not damaged 

is The information of the copyright embedded within the moving image is not cumbersome during ordinary look- 

ing and listening. 

A distortion or the degradation of an image quality is not afforded to the moving image, or at least it is not 
noticed by the user. 

(c) Even when the moving image has been partially pirated, the information of the copyright remains in the pirated 
so part. ~ 

The copyright information is embedded everywhere in the moving image in order that, even in the case of the 
partial piracy, the possibility of the inclusion of the copyright information in the pirated part may become high 

(d) The copyright information in the moving image can be detected without requiring any special device. 

Concretely, when the frame containing the copyright information is viewed as the still image, the copyright infor- 
55 mation is visually perceivable clearly. 

Thus, also the general user (viewer) of the moving image can recognize the existence of the copyright when 
he/she brings the moving image to a standstill or feeds it in frame units in the display mode of a video record- 
ing/reproducing apparatus or a personal computer, or the like. 
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tal „ , ««. whete the moving image is d <fgM type, it is immune egainet operation ascnbable to the com- 
^Z^^^^^^^^^ can <«•* — « ««- 

>m *%SZ£!£L»~r covers the frame poaMons (heme numbe*) th which the cop^ Werma.cn has 
restoring the original moving image before being watermarked, as may be needed. 

formance adopts the following expedients (a) ~ (1): 

(a) The information indicative of the copyright is inserted as an ° f ™™ 9 

Sn'the form in which the informatior , is visuaHy is not too conspicuous. On 

"'' , Stofa«ris^tenTea e ;e hekt the indicating dutatien of the intaga o. .he copyright information vent short. 
Ci "tnan Sfin^ge is too farge. tor exampie. whan i, «a op a screen. . becomes consptcuoos in. a. ... a atow 
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motion mode and will offend the eye of the viewer. Moreover, it is easily identified by a pirate. 

(e) The frames into which the copyright information is to be inserted, are inserted at irregular intervals. 

Thus, any person who intends to plagiarize the moving image data cannot readily conjecture the insertion posi- 
tions. 

(f) In the above item (e), for selecting the insertion frames by the use of pseudorandom numbers or the like , it is 
recommendable to employ a method which selects the positions at random and which is controlled. 

The method has the merits that the insertion of the copyright information is automatically effected, and that the 
numbers of the insertion frames can be readily reproduced with a small number of items of control information, for 
example, an algorithm for random number generation and a seed used therefor. 

(g) Those positions in the insertion frames at which the copyright information is to be inserted should not be fixed, 
but they should preferably be selected at random. 

This is also intended to make it difficult for the pirate to know the insertion positions. 

(h) In the above item (g), for determining the insertion positions by the use of pseudorandom numbers or the likes, 
it is recommendable to employ a method which determines the positions at random and which is controlled. 

The method has the merits of automation and reproducibility. 

(i) In inserting the copyright information, the creator of the moving image content records the insertion frame num- 
bers, the insertion frame positions, insertion information, etc., in order that the recorded items may be utilizable in 
case of legally disputing about a copyright problem in the future. 

The fact that a person can point out the places of the information items inserted so as to be usually unnoticed 
and to be random, without inspecting all the frames of the content, becomes powerful evidence for proving that 
he/she is the creator, or at least, inserter of the information. 

G) The above records may well be owned in the form of the list of the individual data. Alternatively, in the case of 
employing the expedient (f) or (h), it suffices to record only the small number of items of control information, 
(k) In inserting the copyright information, the creator can record those parts of the original image which are to be 
hidden by the information. 

Using the records of the partial images and those of the above expedient (i) (or the item (j)), the creator can 
restore the original moving image into which the copyright information has not been inserted yet, as may be 
needed. 

(I) Although the information items of the above expedient (i) (or fl)) (and (k)) can be kept confidential by the creator, 
they can also be borne in the watermarked moving image after having been encrypted and be circulated by the 
same storage medium. In this case, only persons who have keys for decrypting the codes can decipher the infor- 
mation items of the expedient (i) and know the insertion positions of the copyright information. 

[0052] Besides, only the persons who have the keys for decrypting the codes can decipher the information items of 
the expedients (i) and (k) and restore the original moving image into which the copyright information has not been 
inserted yet. 

[0053] In the above, there have been described the conditions which the watermark insertion device of the present 
invention is to fulfill, and the expedients which implement the conditions. The practicable aspects of performance of the 
conditions will be described in detail below with reference to the drawings. Incidentally, the ensuing description shall 
exemplify a case where the original creator who has created and edited the contents of a digital moving image by the 
use of the moving image processing software of a personal computer asserts the copyright of the data. The creator 
inserts the information of copyright indication as a visible watermark into the digital moving image in order that the 
inserted watermark may be immediately detectable even when the digital image has been plagiarized. 
[0054] Fig. 5 is a flowchart showing the basic flow of a watermark insertion process in the present invention. By the 
way, in flowcharts to be referred to below, a substitution shall be expressed by ":=", and a comparison by *= M . 
[0055] Referring to Fig. 5, when the process has been started, the watermark to be inserted into the original moving 
image is first selected as a two-dimensional pattern at step S1 . At step S2, the inter-frame positions, that is, the frame 
numbers of frames into which the watermark is to be inserted are determined using a pseudorandom number genera- 
tion function. 

[0056] When the frame numbers (11 ,12, • • • ) of all the frames into which the watermark is to be inserted have been 
determined at step S2, positions at which the watermark is to be inserted in the respective frames, in other words, the 
coordinate positions of the left lower points of the pieces of the two-dimensional pattern selected at step S1 , are deter- 
mined using a pseudorandom number generation function at step S3. When the positions ((X1, Y1), (X2, Y2), • • • ) of 
the left lower points of the pieces of the two-dimensional pattern in all the frames to have the watermark inserted there- 
into have been determined at step S3, the two-dimensional watermark pattern pieces are superposed on the sampled 
value areas of the original moving image (that is, considering that the image is a set of sample points) on the basis of 
the information items about the insertion frame numbers and insertion positions at step S4. 

[0057] Fig. 6 is a diagram for explaining a method of determining the frame numbers at step S2 shown in Fig. 5. Refer- 
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ring to Fig. 6. the pseudorandom number generation function 15 outputs tine frame, numbers of the « into which 
nexf^me numoer * the s'eed outputted wrth the frame number is simi.arly used for outputting the frame 



moT "tZ "Soupu'tting of the frame numbers by the pseudorandom number generation function 15 will be con- 
^f? J^n^H St aiumed bv way of example that the moving image continue for about 30 mmutes as a whole 
cretely explained. It ,s f^ um ^£ w Z ooo frames in total Although the total number N of the frames may well 

b— nTifa^ 

? S K " . a SndomsoLtthe copyright information may be inserted ata rate of one 

moving image on the average, that is, at a rate of one time in n/30 seconds. Here tine value n shall be set at 1 50 by way 
of examole Thus the copyright information is inserted into one frame in 5 seconds. 
SSsTa?^ of the insertion can be contrived so as to be determined by, 

SLhv the Thole moving image is divided at intervals of a predetermined time penod, and tiie rate of the insertion 
fs cSSby the ^random number values in the respective time intervals. Byway of example, the whole moving .mage 
S^SI^Son. minute, and the rate of the insertion is changed using jh. «r*m ^^"jg 
a manner that "n1 " frames are inserted in the first time interval of one minute and that n2 frames are .nserted 

20 ^TtZZlTe^Lr* value of the "nx" values is set within a range in which any change ascribable to the 

decree orcomplexity of the selection of the insertion positions of the watermark he.ght.ens more, and a th,rd party f .nds 

iTS T^^SS a^u^A^ich generates uniform random numbers (real numbers) within a range 
57?1 ZSo^\^ g 1 isemployed as the pseudorandom nur^ergenera^^ 
30 dom number generating seed S[0] is designated as the seed for determmmg the f ,st frame number 11 On th.s occa 
sion. the f irst frame number is given by the following equation: 

11 = floor(RANU(S[0]) x n) + 1 

rniwsi As stated before the function RANU outputs simultaneously with the first frame number 11 . the seed S[1] which 
sTsed foTgenefatXe next frame number .2. Here in the above equation "floor" denotes a function wh.ch discards 
the value of an argument in parentheses, below the decimal point thereof, .nto an integer. 
[00641 The kth frame number Ik is generally given by the following equation: 

Ik = floor(RANU(S[k-1]) x n) + 1 + n(k-1) 

rnnesi Here the last term "n(k-1 )" indicates that the Ikth frame as which one frame is selected every 1 50th frame 
SSL to the ktTone of seTs each consisting of 1 50 frames, among 54.000 frames in total. The function RANU outputs 
t^se* f SM saving n* frame number, together wrth the random number for determining the frame 

45 SoSr Various methods are considered for calculating the random number generating seed S[k]. By way of examp.e. 
as follows: 

Initial seed X[0] (optionally selected for 0 < X[0] < m - 1) 
[0068, Next seed X[n + 1] = a ; X[n, jbjjodnj Here. the ^ of |^ ™ 

^^^^^ 5 ul^r bating the seeds. The methods are bu, in the random 
SL generation function, and are prepared as mathematical .ibraries in most programming languages. 
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[0070] Fig. 7 exemplifies the determined results of copyright information insertion frames obtained in the above way. 
As explained before, one frame is selected at random from within each frame set consisting of q frames, and the copy- 
right information is inserted into the selected frame. In Fig. 7, squares coloured black denote the frames in which the 
copyright information is inserted. Within one set of n frames, the frame in which the copyright information is inserted 
5 may lie at any position. Therefore, the possible maximum value of the spacing between the frames in which the copy- 
right information is inserted is (2n - 1), while the possible minimum value is 1 (one). The average value of the spacings 
is n, and the maximum value thereof is guaranteed to be, at most, 2n. 

[0071 ] Besides, unlike the above case where the rate of the insertion of the copyright information is uniform in the 
moving image, it can be contrived so as to differ in the moving image. By way of example, in the case of selecting the 
10 insertion frame from within each set consisting of a frames as stated above, the value n may be changed as random 
number values. Alternatively, the moving image may be sectioned at intervals of a predetermined number of frames, 
and the number of the insertion frames in each section may be changed. 

[0072] Fig. 8 exemplifies the determined results of the insertion frames of the copyright information in the case where 
the rate of the insertion has been changed by the random number values. In Fig. 8, as in Fig. 7, squares colored black 
75 denote the frames in which the copyright information is inserted. In the example of Fig. 8, the number of the frames in 
which the copyright information is inserted, within each predetermined section t, changes to be 3, 2, 2, 1 , • • • from the 
head at the rates determined by the random number values. 

[0073] When the frame numbers of the frames into which the copyright information is to be inserted have been deter- 
mined, the positions at which the copyright information is to be inserted in the frames having the determined numbers, 

«%o in other words, the coordinates of the lower left points of the pieces of the two-dimensional pattern are subsequently 

J determined using the pseudorandom numbers. 

[0074] Fig. 9 is a diagram for explaining a method of determining the insertion positions of the copyright information. 
Referring to the figure, a pseudorandom number generation function 1 6 is a function which is similar to the pseudoran- 
dom number generation function 15 shown in Fig. 6, and which outputs the intra-frame X-coordinate values and Y-coor- 

25 dinate values of the lower left points of the pieces of the two-dimensional pattern. As in the case of Fig. 6, in outputting 
the X-coordinate value X1 within the first frame, a seed SX[0] is required for generating a random number. Besides, a 
seed SY[0] is required for outputting the Y-coordinate value Y1 . These seeds shall be afforded from outside. 
[0075] As in the case of Fig. 6, in outputting the coordinates X2 and Y2 of the lower left point of the two-dimensional 
pattern within the next frame, there are used seeds SX[1] and SY[1] which the pseudorandom number generation func- 

30 tion 1 6 outputs simultaneously with the outputting of the random numbers used for the determination of the coordinates 
X1 and Y1 . Thenceforth, the X- and Y-coordinates of the lower left point of the two-dimensional pattern and seeds for 
calculating the coordinates in the next frame are similarly outputted. 

[0076] Fig. 10 shows a practicable example of the copyright information. Regarding the copyright information to be 
inserted, an area enclosed with a dotted tine in Fig. 10 is handled as the two-dimensional pattern, which has a size of 
35 Lx dots (in width) x Ly dots (in height), for example, 300 dots x 60 dots. The color and tonal density of letters constituting 
the pattern are assumed fixed for the brevity of description. 

[0077] Fig. 1 1 is a diagram for explaining the insertion of the copyright information into one frame. In the figure, it is 
assumed that one frame have a size of Mx dots (in width) My dots (in height), and that the position of any dot be 
expressed by X- and Y-coordinates with reference to the lower left point of the frame. 



jo [0078] As explained in conjunction with Fig. 10, the two-dimensional pattern as the copyright information to be 
inserted has the size of Lx dots (in width) Ly dots (in height). In order to set the left lower point as the insertion position 
of the two-dimensional pattern, the range of the insertion position in one frame is limited within an oblong the two sides 
of which are indicated by a dotted line in Fig. 1 1 . That is, the range extends from 1 to (Mx - Lx) in terms of X-coordinate 
values and from 1 to (My - Ly) in terms of Y-coordinate values. 
45 [0079] The pseudorandom number generation function 16 shown in Fig. 9 outputs the coordinates X1 and Y1 of the 
left lower point of the two-dimensional pattern by the following equations, in correspondence with the respective values 
SX[0] and SY[0] afforded as the random number generating seeds to the first frame as explained before: 



[0080] Here, regarding the functions RANU, the same function as in the foregoing case of outputting the frame num- 
bers is used. Simultaneously with the outputting of the coordinates X1 and Y1, the pseudorandom number generation 
55 function 16 outputs the random number generating seeds SX[1] and SY[1] for outputting the insertion position of the 
two-dimensional pattern in the next frame. 

[0081] The coordinates Xk and Yk of the position at which the two-dimensional pattern is to be inserted, in the kth 
frame into which the copyright indication is to be inserted, that is, in the frame of frame number Ik, are given by equa- 




X1 =floor(RANU(SX[0]) x (Mx - Lx)) + 1 



50 



Y1 =floor(RANU(SY[0]) x (My - Ly)) + 1 
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tions below. The random number generating seeds SX[k] and SY[k] which are used for oulputf ng the insert f ° n ^ ti ° n 
of the two-dimensional pattern in the next frame are outputted from the pseudorandom number generate function 16. 
together with the random numbers for computing the positional coordinates. 

Xk = floor(RANU(SX[k-1]) x (Mx - Lx)) + 1 



Yk = floor(RANU(SY[k-1]) x (My - Ly)) + 1 

r00821 Has 12 and 13 are flowcharts showing the detailed flow of the visible-watermark insertion process, with the 
Lbole description summed up. Referring to Fig. 12. when the process has been started, a file F .n wh.ch a moving 
fmagefo S nVndli. namely an original moving image not bearing an inserted watermarkisstored.sf.rst accessed at 
sTep ll 1 ft is assumed that fie frame size of the image be Mx dots * My dots, and that the total number of frames be N. 
[00831 At next step S12. copyright information to be inserted as the watermark is created as an .mage, namely, a two- 
dWsional patten? The size of the pattern is assumed to be Lx dots x Ly dots. Subsequently, at s ep S1 3. control 
tSSSSZ watermark insertion are set and are stored as control information. Set as the control parameters are 
ihe average interval q of the frames into which the copyright information is to be inserted a random ~^fl™£"° 
seed SIOl for determining the number of the first frame, random number generating seeds SX[0] and SY[0] for deter- 
mining 5 a^Wdinate values in the first frame, and a function RANU for generating uniform random numbers .n a 

££]* Xtte 'control parameters have been set at step S13. the determinations of the number of the frame into 
wSch the copyright information is to be inserted and that position in the frame at which the copynght .nformaf on .s to 
btfnsert^S of the copyright information, etc. are iterated at steps S14 - S17. At step S14. the number of 
fhe frlmeTnto which the copyright^nformation is to be inserted is determined as explained before. Step S15 shown ,n 
Fig *S de^es the inserton position of the two^imensional pattern as the copyright 

mined frame namely, the coordinates of the left lower point of the pattern within the frame. Subsequently, at step S16. 
S pari Z the Original -age into which the two-dimensional pattern as the copyright nformafon ,s to , be .nserteo. 
namely the part which is designated by four vertices is duplicated and has its content retamed m a separate f.le FD .n 
o^atthe origina, image before the insertion of the copyright information may be restored at need. Thereafter, at 
steo S1 7 the image of the copyright information is duplicated or overwritten in the designated area. 
[00^] After7e process of steri S14 ~ S17 has been iterated N/n times (k > 1 ~ N/n ). the waterrmrk .nformaton 
anTme confro. parameters thereof are stored in a separate file FC at step S18. Then, the watermark ;»>rt°npo~ 
fs ended Here contents to be stored as the control parameters are Mx and My ind.cat.ve of the s.ze of each frame and 
N frScative of the total number of frames as are the information items of the moving image hie F; Lx and ^ ."d'catve 
of the size of the image of the watermark and the data of the image itself; the average frame .nterval or spacmg q of the 
waVemUk insertion; the number N/n of the insertion frames; the pseudorandom number generat.onfunct.on RANU. the 
se2s sTo] S?[0] and SY[0] for generating the random numbers; and the calculating algorithm for the frame number Ik 
and insertion position (Xk, Yk) as stated before. 

5S5T Aftef having inserted the copyright indication in the above way. the creator of the moving >^«««* 
eoords in the form of a list as one method in order to record the insertion posit.ons of the copynght .nd.cat.on within the 
image. More specifically, the list can be created with each set consisting of the frame "umber k end the ^bonpos.- 
t™ (Xk Yk) for all the values k :« 1. 2. • ■ • , and N/n. As a more compact record.ng method, the control .nformafon 
items which are the control parameters mentioned at step S1 8 in Fig. 13 can be recorded and retamed. 
[W»87] Fig. 1 4 is a diagram for explaining an example in which such control informat-on terns are ™£™ 
n the moving image file. Referring to the figure, besides frames of watermarked moving .mage in all he watermark 
conTdToTmaf on items which are the control information items mentioned at step S18 in Fig. 13 and al. of which have 
DeT e rypted are stored in the control table of the file. In a case where the watermark control ^^JT^ 
stored in the control table of the file in this manner, and where they are duplicated .n. for example, afloppy d.sk together 
S -Sk Tmoving image, the control information items can also be offered directly to the user ede. Since. 
Z STSSlTwiEmSk control information items have been encrypted, a user having a key for decrypf on can know 
he contems of the waternna^ control information items by utilizing the key, whereby he/she can know the .nsert. on £os, 
ionf^e copyright information. On this occasion, when the partial images of the or.g.nahmage .before the .nserf on 
of the copyright informaf on as duplicated in the separate file FD at step 818 are offered ^.^^St 
encrypteTstates. a user having a key for decryption can also restore the original moving .mage .n wh.ch the copynght 
informaf on as the watermark has not been inserted yet, by utilizing the key. 

mSl | Fig. 15 is a flowchart showing a process for restoring the watermarked moving image .nto he ong-nahmage 
which is not watermarked, in this manner. Referring to the figure, when the process has been started. ^ set of the 
moving image files of the original copyright holder are accessed at step S21 . The set of the f.les mcludes ttie . contente 
of a watermarked moving-image file FA. the file FC storing the watermark control informaf on therem. as well as the con- 
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trol information which is as mentioned at step S18 in Fig. 13, and the file FD in which the parts of the original image 
hidden by the watermark are saved. 

[0089] Subsequently, the value of the variable k indicative of the number of the watermarked frame among the group 
of frames in which the watermark or copyright information is inserted is initialized to 1 (one) at step S22. Thereafter, 
processing at steps S23 - S26 is iterated. At step S23, the frame number Ik and the insertion position (Xk, Yk) of the 
copyright information are first computed in the same manner as in the processing at step S14 in Fig. 12 and step S15 
in Fig. 13. At the next step S24, the original image in which the copyright information is not inserted is restored for the 
frame Ik by the use of the saved partial image. In this processing, the partial image retained in the saving file FD is dupli- 
cated at the watermark insertion position which is determined by four vertices. 

[0090] Subsequently, whether or not the value of the variable k has reached N/n is decided at step S25. On condition 
that the number of times N/n has not been reached yet, the value of the variable k is incremented one (+1) at step S26, 
whereupon the processing at step S23 et seq. is iterated. Upon the decision of step S25 that the variable k has reached 
the number of times N/n, the original-image restoration process is ended. 

[0091] As described above, according to the present invention, the watermark information or copyright information 
which is visually perceivable clearly in the static or still image is inserted into the frames of the moving image at irregular 
intervals. However, in the case where the copyright information is viewed in the moving image, it cannot be visually rec- 
ognized because the indicating duration thereof in one frame is sufficiently short. The information which is inserted in 
this way, shall be termed the "visible watermark" in the present invention as stated before. Such an effect has already 
been extensively known, and has been psychologically researched in. The present invention utilizes such an effect pos- 
itively for copyright protection. 

[0092] Incidentally, as is known, notwithstanding that the information inserted in the form as described above is not 
visually recognized definitely, it is subliminally memorized by the viewer. Therefore, the applications of the effect to 
advertising etc. are thought outrages against human rights and are legally regulated. This point needs to be taken into 
consideration even in a case where the visible watermark is utilized only for the copyright indication in the present inven- 
tion. 

[0093] Fig. 16 is a diagram for explaining a copyright indication method which serves to exclude the influence of the 
subliminal effect. Referring to the figure, the information of copyright indication being a visible watermark is inserted in 
a frame image being the body of a moving image, at irregular intervals (as denoted by black squares). Moreover, the 
copyright information is indicated at the opening and end of the moving image, for example, a cinema for time periods 
of, for example, several seconds so as to be definitely seen by a viewer. Thus, the copyright information which is mem- 
orized on the basis of the subliminal effect becomes the same as the one which is definitely indicated at the opening 
and end, so that any bad influence on the viewer is avoidable. 

[0094] While the whole aspect of performance of the present invention has been detailed in the above, the present 
invention is not restricted to the foregoing aspect of performance, but it can be carried out in various different aspects 
of performance. In the foregoing aspect, one sort of information as shown in Fig. 10 has been mentioned as the copy- 
right indication. It is also possible, however, that a plurality of sorts of copyright indication information be prepared, for 
example, in English and Japanese, and that at least two of the plurality of sorts of information be selected in accordance 
with designations based on pseudorandom numbers and be inserted into the frame image of a moving image data, in 
the same way as in the operation of selecting the numbers of the frames to have the copyright information inserted. 
[0095] Moreover, the information of the copyright indication is not restricted only to the letters as shown in Fig. 1 0, but 
it can contain a pattern. Such examples are illustrated in Figs. 17 and 18. Fig. 17 illustrates the example in the case 
where any of different patterns (which differ depending upon the number and positions of black squares) is inserted as 
part of the copyright indication information in accordance with designations based on pseudorandom numbers, in the 
same way as in the case of selecting the two or more sorts of copyright indication information. The degree of complexity 
of the copyright indication can be heightened by inserting the different patterns in this manner. 

[0096] Likewise to the example of Fig. 1 7, the example of Fig. 18 is such that any of different patterns is inserted as 
part of the copyright indication information. By way of example, the different patterns to be inserted are obtained in such 
a way that the frame numbers of frames to bear the copyright indication information are encrypted using a hash function 
or the like. Thus, the copyright indication information itself is encrypted, and the degree of complicacy of the indication 
is heightened still more. In addition, when a technique which allows decrypting the frame numbers from such patterns 
conversely is employed for the above encryption, it will be effective in cases of detecting the copyright indication infor- 
mation and restoring an original moving image. 

[0097] Regarding the frames into which the information of the copyright indication is to be inserted, it has been 
described in conjunction with Fig. 7 that only one frame is randomly selected from within each group consisting of n 
frames. However, such presentation in only one frame is not restrictive, but the copyright information can also be pre- 
sented in a small number of successive frames, e.g., two or three successive frames. The numbers of the successive 
frames can be selected by designations based on pseudorandom numbers, in the same way as in the foregoing. Owing 
to such presentation in the plurality of successive frames, in case of employing the moving-image compression scheme 



13 



EP 0 899 688 A2 



10 



15 



20 



25 



30 



35 



40 



45 



50 



sTc^rr^.^ 

0, so as to besoms me ^gn, indication can bs inserted Wo lbs mo™* image as 

K U .^s^^e^StZ». anoU ajso ba easii, automated using d,g«.morfng.,mage 

editing tools built in a <*^'J°™?*!. _ m iMica4c „ mm, te contained in tbs moving imags created 

ss ^ssksssss » XssBasscss s= sit: 

DW. 



below. 



it is somewhat uneasy. 



55 



14 



EP 0 899 688 A2 



a situation where the copyright indication of the copyright holder A remains in the moving image of the commodity of 
the dealer B. 

[0106] In a case (b2), the frame Ik in which the copyright indication of the copyright holder A is inserted is skipped. 
[0107] In a case (b3), the part of the copyright indication of the copyright holder A is overwritten by the copyright indi- 
cation of the dealer B, and the copyright indication of the copyright holder A is erased. 

[0108] In a case (b4), the image of the frame number Ik is created by an interpolation from the preceding frame of 
frame number "Ik - 1" and the succeeding frame of frame number "Ik + V. 

[01 09] In the cases (b1 ) and (b2) among the four cases, it is objectively persuasive that the copyright holder A directly 
asserts the piracy by the dealer B. In the cases (b3) and (b4), however, the conclusion of the piracy is delicate, and a 
judgement by an expert who is a public third party will be necessary. 

[01 1 0] The copyright holder A can request the expert of the public third party, for example, the Authentication Office, 
to give an expert opinion on the piracy. Expedients for the judgement are basically the same as in the cases (b1) ~ (b4). 
The expert of the Authentication Office views the images themselves to decide if the watermarked original moving 
image of the copyright holder A and the moving image of the commodity of the dealer B have an identical source. Upon 
deciding the identical source, the expert judges which of the images is piratic. In filing a petition with the Authentication 
Office, the copyright holder A submits the control information of the insertion of the watermark as mentioned at.step S1 8 
in Fig. 13. Also, the dealer B can submit similar control information (if any) as a counteraction. As a result, the following 
cases can take place: 

(d) The copyright indication of the copyright holder A has been found in the moving image of the commodity of the 
dealer B. Then, the plagiary of the A's moving image by the dealer B is proved. 

(c2) The moving image of the commodity of the dealer B includes the part of which the preceding and succeeding 
frames contain unnatural movements, and which is decided to have come from the removal of one frame. Moreo- 
ver, the part corresponds to a frame in which the copyright holder A has inserted his/her copyright indication. Then, 
the plagiary of the A's moving image by the dealer B is proved. 

(c3) The places of the copyright indication inserted by the dealer B are arranged so as to cover up the A's copyright 
indication without fail. If the places of both the A's and B's copyright indications match completely, which of the mov- 
ing images is the source cannot be immediately concluded without further data. In general, the moving image in 
which the indication is presented in a small number of places has a higher possibility of being the source. 

In this case, the decision becomes quite definite when the copyright holder A is let submit the control informa- 
tion of the watermark insertion. The copyright holder A reports the use of the pseudorandom numbers in the pres- 
entation of the copyright indication, and demonstrates the value of the seed of the used random numbers, as well 
as the algorithm for the generation of the random numbers. Thus, the copyright holder A can automatically repro- 
duce all the insertion positions of his/her copyright indication. In contrast, the dealer B is unable to submit such sys- 
tematic materials. At most, the dealer B can merely demonstrate concrete values at the piratic parts, in the form of 
a list. This is because no method other than trial and error can be used in order that the seed value employed for 
the generation of the uniform random numbers may be conjectured from the output results of the random numbers. 
(c4) In a case where the dealer B has produced as their commodity the moving image in which the part of the cop- 
yright indication of the copyright holder A has altered by a method such as the interpolation from the preceding and 
succeeding frames, the decision becomes more delicate, and the naturalness of the image is first disputed about 
in the same way as in the case (b4). 

[01 1 1 ] In this case, for showing the dealer B to have copied, the copyright holder A will need to submit to the Authen- 
tication Office the control information of the watermark insertion and the original moving image before the watermark 
insertion. Then, it can be proved that the portion of the moving image posing the problem of the piracy by the dealer B 
is part of the moving image of wider range lawfully owned by the copyright holder A. More specifically, the parties A and 
B own the portions of the moving images decided to come from an identical source, and the portion owned by the party 
A is larger than by the party B. Herein, if the legality of the copyright of the holder A has already been established, the 
assertion of the dealer B owning only the partial image weakens. By the way, it is also considered in this case that, after 
the moving images of the parties A and B coming from the common source were separately distributed, the party A has 
inserted the copyright indication, while the party B has also gained the lawful copyright of the moving image in any form 
and has put the moving image into the article of commerce. 

[0112] As thus far explained, the decision of the piracy is quite definite in any of the cases (a), (b1), (b2), (c1), (c2) 
and (c3), and the piracy by the dealer B is socially clarified with ease. The most delicate case is the case (c4). However, 
it requires enormous labor for the dealer B to alter the moving image created by the copyright holder A, to the extent of 
withstanding such an expert opinion. More specifically, the dealer B needs to execute such a high degree of moving 
image processing that the portion of the A's moving image to be pirated is fed every frame so as to find out all the pieces 
of the A's copyright indication, that each part of the copyright indication is restored by the method such as the interpo- 
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lation from the preceding and succeeding frames, and that the image is ^^^^T^^T^e 
Se" w^eTm™^ watermarked moving image and the confdentia. or encrypted 

nrrasion the kev of the encryption is registered in a sealed state. 

[01 1 3 I ^manner, the labor and camouf .age which the dealer B requires for the piracy of the = e MJJJ £ 
not wumerbalance the i prof its. In this sense, the visible watermark according to the present ,nvent,on faahtates the 
2vl~nrT of the oiracv of the moving image and can therefore prevent the piracy from occurnng. 
SET ass^in 9 and verifying the legality of the copyright against the piracy of the mov,ng .mage as 

deale B^out tn^ legS y of the copyright of the moving image. In this example, the dealer B has pirated only part of 
Z ^n^g?ofTe tewfu copyrighfholder A. and the copyright indication of the copyright holder A inc uded .n the 
cirrTSorSing ^ has been ovenLSen by the copyright indication of the dealer B. thereby to be ^ Inth.s casj 
♦Kl?nSI hoWer A can reveal the numbers of the frames bearing the inserted watermark and the control pa ameters 

mark insertion positions within only the part at issue. Accordmgly, rt can be clearly proved that the legality ot tne copy 

LecS pS' Referring to the figure, when the process has been started, the set of the mov,ng .mage Wes of th e o r.g- 
pectea piracy. nererr y J* t & S30 Data t0 be accessed here are the same as mentioned at step S21 in 
Ro «t£ I they ar^e^eSs of Te watermarked moving-image file FA. the file FC storing the watermark ^n- 
trolinformaton^ 

T^^^Ts^W. the .ist of watermark insertion positions is created using the watermark control 
formation oh2 <^ cwyright holder A. and the resulting list is stored. In creating the list, the frame numbere Ik 
^T^IiS oHh e : watermark insertion positions in the watermarked frames are computed for the 

fram^m^ 

?01 IsHf nexS S. the moving image file FB of the dealer B suspected of plagiary is acceded so as to find the 
S^L framed ; oTaYusoected plagiarized part in the moving image of the dealer B. At step S33. a frame IA in £e 
nT nl ! toSU holderA as corresponds to the above part is searcher so , . to co^rm ^ 
corresponding relationship between the frames IB and IA. Thereafter, processing at steps S34 ~ S37 in Fig. 22 ,s 



35 

ated. 



rniOl At step S34 the number of the first frame bearing the inserted watermark information is obtamed from within 
SI mLino imSe of the copyright holder A beyond the frame number IA. and the images of the obtained frame number 
40 the ArmSimage andTcorrespondentframe number in the Bs moving image are compared by ^g the ^ 
mlnn imTarandThe Bs moving image of suspected plagiarism in frame feed modes. In the companson, the contents 
XSfSn JSS >ll pa^of the original moving image not watermarked yet are stored and which ,s kept m cus- 
todv on the side of the original copyright holder A are also utilized as may be needed. 

0I 20T Subsiuentiy at step S35. the result of the comparison between the As image and Bs ,mage is judged as to 
45 Kon (a? LTerand judgement is recorded. The situation (a) is a case where the watermark of the copyright 
ho^A remain! in the B s image, while the situation (b) is a case where the frame of the As moving .mage 1 atissue .s 
skiooed in X moving image. The situation (c) is a case where, in the Bs moving .mage, the copynght .nd.cat,o of 
^ cSvnaW loldeT A is overwritten by that of the dealer B in an erasing form. In the situation (d). the copyright .nd,ca- 
ton 7£ ZSSSt Cofder aTs not extent in the image B. and the part is close to the original image of the copynght 
«, holder ? bSS watermark insertion, but it is somewhat unnatural. In the situation (e), the .mage of the frame in the 
Sf££5 lg?nS^mpSe.y wrth the original image of the copyright holder A before the watermark .nserhorK 
f 0 l 2 T When the result has been recorded at step S35. whether or not similar images are cont.nu,ng in the two As 
Edil moving mages is decided at step S36. Subject to the decision that the similar .mages are continuing, the value 
oHhe vahaWe k^ incremented at step S37. Thereafter, the processing of step S34 et seq. is iterated. 
55 mM^^VtadStan at step S36 that the similar images are not continuing in the two, As and B's moving 
maaes the S t ,Ttoe comparisons at step S35 are synthetically judged at step S38. As a conc.us.on 1). when the 
caseta) eleven * once, it is synthetically judged that the suspicion of plagiary by the dealer B .s confirmed. 
[0123] S fusion (2). in a case where the resurt (b) or (c) has been obtained in all the comparisons, the susp, 
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cion of plagiarism is judged almost confirmed. 

[0124] As a conclusion (3), in a case where the result (b) or (c) has been obtained in most of the comparisons and 
where the resuft (d) or (e) has been obtained in some of the comparisons, the suspicion of plagiarism is decided very 
certainly. 

[0125] As a conclusion (4), in a case where the result (b). (c) or (d) exists in some of the comparisons, the suspicion 
of plagiarism is decided strong 

[0126] As a conclusion (5), when all the comparisons has fallen under the case (e), it is decided that the dealer B 
would have obtained the original moving image in which the watermark was not inserted by the copyright holder A yet, 
or that the dealer B would have perfectly restored the original image not bearing the inserted watermark yet, by any 
method. 

[0127] Fig. 23 is a diagram for explaining examples of the results of the synthetic decisions which are rendered at step 
S38 of the flowchart in Fig. 22, in correspondence with the judged results at step S35. As shown in Fig. 23, in the case 
where the judged result (a) has been obtained as to all the frames of the suspected plagiarism part or even if in one of 
the frames, the synthetic decision is the result (1). In the case where the judged result (b) or (c) has been obtained as 
to all the frames, the synthetic decision is the result (2). In the case where the judged result (b) or (c) has been obtained 
in most of the frames and where the judged result (d) or (e) has been obtained in some of the frames, the synthetic deci- 
sion is the result (3). In the presence of the judged result (b), (c) or (d) in some of the frames, the synthetic decision is 
the result (4), and in the presence of the judged result (e) in all the frames, the synthetic decision is the result (5). 
[01 28] Thus far, the insertion of the copyright indication being the watermark information has been described in detail. 
Now, there will be described the insertion of copy restriction information into a moving image, and the video reproduc- 
tion and recording of a moving-image storage medium bearing a recorded copy restriction code. 
[0129] As stated before, the insertion of the copy restriction information complies with the Call for Proposals issued 
by the CPTWG/DHSG, and the basic idea of the present invention in this regard becomes as indicated by the following 
items (10): 

(1) The concept of the DHS watermark according to the CPTWG/DHSG as explained before is fundamentally bro- 
ken up, and two sorts of information are employed. 

The first is information indicative of a copy control mode. This information is not handled as a watermark, but 
it is set as definite control information which is processed by hardware. 

The second is information for presenting copyright. This information is handled in the state in which it is put in 
moving image data as a watermark. The watermark need not be detected for the control of copying or duplication. 

It is the basic idea of the present invention to conjointly use the two sorts of decomposed information. 

(2) The copy control modes include the three kinds of modes in the Call for Proposals according to the 
CPTWG/DHSG, and are not restricted to these three kinds of modes. At least, an "any-copy mode" which permits 
duplication any number of times is added to establish at least four kinds of copy control modes. These four kinds of 
control modes are illustrated in Fig. 24. 

(3) As the extension of the copy control modes, it is also possible to introduce any of a contrivance for restriction 
based on the number of generations of duplication, a contrivance for restriction based on the number of children in 
duplication, and a contrivance for restriction in which the first and second contrivances are combined. 

(4) The information indicative of the copy control mode (termed the "copy control code") is stated (stored) in a con- 
trol information area for the whole moving-image file and a control information area for each frame or a group of 
frames constituting the moving image. In a medium drive such as video recording/reproduction apparatus, the copy 
control code is directly processed hardware-wise. Thus, the copy control mode is reliably detected and entered in 
a video recording operation, and the general viewers can be prevented from pirating the moving image data. 

(5) A medium which is capable of video recording and which has a write protection area, includes a data area 
into/from which an image can be recorded/reproduced by the ordinary user, and the protected control data area 
into which a vendor writes data initially and which is processed only hardware-wise thenceforth. In this case, the 
copy control code is written into the protected control data area and is protected from the user. 

(6) In a case where the information for presenting copyright is inserted as the watermark, the scheme of the visible 
watermark as described before is completely adopted, and the visible watermark is inserted into the data in the 
data area of the moving image. 

(7) Further, in order to intensify the prevention of piracy, a watermark camouflaged so as to be difficult of visual per- 
ception (that is, an "invisible watermark") may well be inserted into a plurality of frames selected as desired, in addi- 
tion to the visible watermark. As the watermark which is additionally inserted in this case, one of a type which is 
written in a spatial region is more desirable than one of a type which is written after conversion into a frequency 
region, from the viewpoint of easy processing. 

(8) The visible watermark written in the data area of the moving image (including also the additionally inserted invis- 
ible watermark in the preceding item (7)) may be checked only when piracy is suspected. In the usual video record- 
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compatibilities are as stated below: 
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tent of the new scheme according to the present invention, the copy control information and 
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W™^u"1Sds of copy cont™. nrodes shown in Fig. 24 „ ino.rna.od. and recording parrnissiorVbarring or 
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reproduction permission is given in accordance with specifications shown in the figure, in response to a request for 
video recording or a request for reproducing display. 

Especially in the one-copy mode, a child (a copy) can be formed, but the formed copy is set in the no-more- 
copy mode, so that a copy corresponding to a grandchild cannot be formed. In each of the no-more-copy mode and 
the never-copy mode, the reproducing display is permitted, but the video recording is rejected. It is accordingly 
avoidable to form a large number of copies unlawfully. 

(c) The any-copy mode has been added as a new copy control mode, and it has been established as one of the 
formal or regular modes in the new scheme of the present invention. This mode imposes no restriction on duplica- 
tion or copying even in an apparatus or a processing method conforming to the new scheme, and is equivalent to 
the fact that video recording is not rejected in conventional video apparatuses. The effect of the formal recognition 
of this mode is the realization of the compatibilities between the new scheme and conventional schemes. More spe- 
cifically, at present, there are large quantities of moving image packages and contents based on the conventional 
schemes for video recording, the video apparatuses of the conventional schemes have been extensively spread in 
homes and offices, and broadcasts and communications of the conventional schemes are proceeding all over the 
world. The present invention provides a contrivance according to which, in such a situation, the new scheme is 
spread into the market without confusion, and the generations of the schemes are altered surely even though 
slowly. 

(d) The copyright holder of an original moving image can determine if he/she allows his/her customers (such as 
users and viewers) to duplicate the moving image, and can designate his/her will within the medium of the moving 
image or within the data stream of broadcast or communication in the form of the copy control modes. The will shall 
be esteemed by apparatuses which adopt the new scheme of the present invention. 

(e) As the extension of the copy control modes, it becomes possible to set a certain limit for the number of children 
or/and to set a certain limit for the generation numbers of descendants. The limits can be set by the original copy- 
right holder, and can be explicitly indicated in the selling/distribution of the moving image. 

(f) The fact that copy control codes are set as explicit control information, not as a watermark, so as to be directly 
handled by hardware, brings forth the effects that the control information can be reliably detected, and that any ille- 
gal manipulation by a user can be excluded. Especially when the control information is written into the storage pro- 
tection area of a storage medium, the user cannot erase or manipulate the control information, and the right of the 
copyright holder is defended. 

(g) In a video tape, broadcast, communication or the like, the copy control code is set every frame of a moving 
image or every plurality of frames thereof. Therefore, the copy control codes are normally detected during the 
traveling/broadcast/communication of the moving image. Even when video recording has been started at any inter- 
mediate part of the moving image, the copy control mode is decided quickly (within one second), and the disposal 
of the permission or rejection of the video recording is taken. On this occasion, the detection of the copy control 
code misses very rarely unlike the detection of the DHS watermark. 

(h) As a matter of course, the copy control codes do not exert any bad influence such as the distortion of a dis- 
played image, in the reproducing display of the moving image, and they do not afford any offensive feeling to view- 
ers. 

Next, the insertion of the visible watermark as stated before has the following features (i) ~ (k) in relation to the 
Call for Proposals issued by the CPTWG/DHSG: 

(i) While the watermark is inserted among the data of the moving image, it need not be normally detected during 
video recording or reproduction for such a purpose as the control of copying or duplication. In this point, the visible 
watermark is essentially different from the DHS watermark of the CPTWG/DHSG. As a result, the processing of a 
video recording apparatus or a receiving apparatus for broadcast/communication becomes very simple, and mov- 
ing image transmission, etc. at high speed can be realized without any problem. 

G) The visible watermark is inserted as part of the moving image data, and it is applicable to all the aspects of uti- 
lization of the moving image. Besides, the watermark does not degrade processing capabilities at all in the trans- 
mission of broadcast, communication or the like and in ordinary video processing (such as video 
recording/reproduction, A-D/D-A conversion, compression/expansion, and encryption/decryption). In addition, the 
watermark itself is robust. 

(k) When the moving image is displayed in a frame feed mode or a slow motion mode, the visible watermark is vis- 
ually perceivable easily and clearly. When a copyright indication is inserted in an evident form, viewers notice the 
copyright indication soon. Unlike the DHS watermark, the copyright indication need not be normally detected, and 
it may be confirmed by such a method only when the ownership of copyright is to be verified. 

[0133] Aspects of performance for realizing the above features of the present invention will be described below with 
reference to the drawings. Fig. 25 is a diagram for explaining the basic operations of a system for inserting watermark 
information and copy restriction information. Referring to the figure, the watermark information is inserted into an orig- 
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medium. On this occasion, the movng .mage ofte " s " b ^J°°^ ^^tto T^he package medium is set on 
MPEG scheme or ft. ''^^ " 
40 EST^SiSi - Sa^aSubs^y. the moving image data are restored in rea. time 
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processes such as decryption and data expans.on. corresponds to a case where the moving image is 

[0140] The third aspect of the, movmc ^as J^.^SSSS forms such as video on demand, tel- 
distributed by commun.cat.on. The ca ^^^^^.^^ A E . mai , annexing. Moreover, communication net- 
so- evision conference, network games. Intern* ^/^^ protocols include many sorts. If 
works include many types such as w.re/rad ,o/mob*e * P * S 'J 1 ™J^^^ 
necessary, the transmitter of the com^ 

processes such as date ^ having received ^ « ansmM&i dat a 

Si0n) ^ . su ^^thomox/inaimaae as shown in Fig. 27 is such that the moving image is stored by the com- 
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sidered a case where the moving image is duplicated for backup in the identical personal computer by a certain user, a 
case where it is transferred to another user and is duplicated in a LAN, a case where it is transferred and duplicated in 
a computer network of wider area, and so forth. It is considered that, in such a case, the moving image will be usually 
transferred and duplicated in any desired format prepared in the data file. It is the actual situation, however, that a com- 
puter user who is an expert having moving-image editing tools can manipulate the moving image data and the copy 
control information. 

[0142] The lower part of Fig. 27 illustrates the processes for the recording and reproduction of the moving image data 
that cope with the various cases of the above distribution aspects. Although the illustration resembles Fig. 26, a read- 
out/reception device 80, a display device 81 and a video recording device 82 coping with the distribution aspects are 
disposed instead of the medium drive 55, the MPEG decoder 56, the display device 57 and the video recording device 
58. Data restored in correspondence with any of the distribution aspects are composed of a watermarked moving image 
68 and copy control information 69, and they are processed by the readout/reception device 80. 
[0143] The moving-image readout/reception device 80 is usually furnished with the reproducing display device 81 
or/and the video recording (storing) device 82, which is/are requested to display or/and record the moving image by the 
user. Herein, the scheme of the present invention features that, in response to the request, the copy control information 
contained in the read-out/received data controls if the moving image is to be actually outputted (to be recorded or/and 
displayed), so as to permit or bar the image output. In particular, in the readout/reception device 80 conforming to the 
scheme of the present invention, the copy control information is processed by hardware, and hence, it cannot be inten- 
tionally altered by the user. Besides, the copy control information is also detected every frame or every group of frames 
in the readout/reception device 80 for the moving image. Therefore, even in such a case where the package medium 
(especially, video tape) has been read out from the intermediate part of the moving image or where the broadcast or 
communication has been received from the intermediate part of the moving image, the output control can be performed 
immediately. 

[0144] As shown on the right side of the lower part of Fig. 27, it is the moving image bearing the watermark information 
(especially, the visible watermark) that is displayed when the display (reproduction) request has been permitted. The 
moving-image readout/reception device 80 does not perform any special process at all as regards the watermark infor- 
mation. It does not detect the watermark information, either. Whether or not the watermark information is borne, the 
device 80 operates to execute the ordinary moving-image display irrespective of the watermark information. Besides, 
the visible watermark affords quite no offensive feeling to a viewer. 

[0145] In a duplicate copy formed by the permission of the video recording, the same watermark information (espe- 
cially, the visible watermark) as in the original moving image is inserted among the moving image data 61 stored in a 
data area. In addition, copy control information 62 for a child copy is written into a control data area hardware-wise in 
accordance with the specifications of the scheme of the present invention. In particular, the no-more-copy mode is writ- 
ten into the child copy accepted by the one-copy mode. 

[0146] What aspect the duplicate copy formed here is in, depends upon the moving-image readout/reception device 
80 which is employed. In the commonest aspect, the duplicate copy is outputted in the form of a package medium for 
the viewer. This aspect corresponds to a case where a video tape storing the moving image therein is duplicated, or 
where a broadcast program is received so as to record the moving image in an MO disk or a DVD. On this occasion, 
the output medium can return to the distribution process of the package medium as shown on the left side of the middle 
stage of Fig. 27. In a higher degree of utilization, there will be a case where the moving image is stored as the computer 
file of a personal computer or the like. On this occasion, the output moving-image storage obtained at the lower part of 
Fig. 27 might be added to a new distribution path in the form of the control-information bearing moving-image data as 
shown on the right side of the upper part of Fig. 27. In this way, the scheme of the present invention can pass any 
desired distribution path or processing path, such as the package medium, broadcast, communication or computer 
transfer, without an intervening fence. 

[0147] Meanwhile, the conventional schemes are considered as that scheme of the present invention in which the 
copy control information is cleared to zero (without being added). Herein, the readout/reception device 80 does not have 
the function of processing the copy control information and barring the user's request (consequently, it permits all the 
requests). The watermark information is not actually borne, either. These facts signify that, even when the conventional 
schemes and the new scheme of the present invention are mixedly used in the whole operation illustrated in Fig. 27, 
any hindering part is not found at all. Heretofore, it has been hardly practiced to restrict copying or duplication hardware- 
wise (this situation corresponds to the any-copy mode according to the present invention as shown in Fig. 24). It is 
accordingly understood that, in the case where the protection of the copyright holder is to be attained by introducing the 
copy control modes as in the specifications of the CPTWG/DHSG (in Fig. 1) or the specifications of the present inven- 
tion (in Fig. 24), the specifications of the CPTWG/DHSG (in Fig. 1 ) are insufficient for spreading the copy control modes 
while compatibilities with extensive conventional systems are held, the specifications of the present invention (in Fig. 
24) being indispensable therefor. 

[0148] Fig. 28 is a table exemplifying copy control codes which serve to realize the copy control modes shown in Fig. 
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is important for compatib. My nlra" but Sey are not a.ways absolute. 

copy mode is encoded as "000" as the purport of the P^JjJlK^ the ^l codes, the copyright indi- 
[01511 The principal control information of the ^^^^^J!^^^ users. Some of package 
cation thereof, etc. shou.d^ 

ISS dTaTTth* £?. ? JlS^*?:.. thfcopy -ntro. cc.es, the copyright indication of the 
original copyright holder, eta into the especially p. a . e J readout/reception device in order to explain how copy 
[0152] F.g. 29 is a block diagrarr . show,ng , the s u^re of ^SSwSEn device 83 includes a data readout por- 
control codes are handled therein. Referring to the figure the readout/ ecep o a control 

tion 84 for reading out moving image SSSSJSrtSf 87 for creating copy moving 

„ ^53] .ncaseofreproducinganddisplayingtheinputt^^^^^^^^ 

to send the reproducing display portion 85 the mow moving image data, the 
reproduce and display the moving image Besides, n ~ *^3^EJ nige data. Herein, on condition 
control code readout portion 86 reads ^^T^t^^^^ the video recording portion 87 
that the moving image data are dupl-cable fte J^^^irMdS portion 84. and it supplies the control code of 

In genera., moving image data have an enormou ^^' n **** information in the head of the 
3 o media include those with which ^J^S^lSSSi SrXage frame is viewed (for example, a mem- 
entire moving image data) ^ be -^^ for the entirety cannot be viewed 
ory in a computer, an MO d.sk, and a DVD and those wnn . , jd t ^ Besides, in a 
when any desired image frame is viewed (for ex a ^ n ^^°^^ ^2e they are being broadcast or 
case where the moving image data^ are image frames are. in substance. 

tion is transmitted at the head part of the transm.ss.on). medjum ^ ^ 

[0155] Therefore, especially in the case of stonng m °3 ma ^ ™ ™ io 7«Xl data such as the copy con- 

sra^^ 

^cTn^a^^^ 

[0156] Fig. 30 is a diagram showing the v.deo fo ; t mat °* althe unit of random access is "GOP", 

of copy control information. Her* tM , mnmum unrt ofa group of frame, > se^ng as t ^ ^ ^ ^ ^ ^ 

and information on a time period ™» e e ^ Extension Data) are prepared in both the 

Data areas for extensions in the future (Seque nee • y inve ntion can be set in the data areas, 
parts SH and GOP. Therefore, ^ co^contro by extending the copy control 
[0157] Next, the extension ^JW^JJ^ * gen eration. and/or to restrict the number (LG) of 
modes to restrict the number (LC) of the duplicate copies or ine cn « speC ifications shown in Fig. 24 cor- 

the generations of descendants. From such viewpoint . the one-copy mode ^n the spea ations is limit ed 

responds to a case where the number of children ^' mrt '^ e -nf ^ of children is limited to 

to one. and the no-more-copy and never-copy modes correspond t«^ ^^ionat ise f ree from the copy restriction (in 
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CCCO 0/1 In conclusion, duplication is permitted (0) or not (1 ) at present 

• CCC1 0/1 The number of children is limitless (the infinite) (0) or not (1) 

• CCC2 0/1 The number of generations is limitless (the infinite) (0) or not (1) 

[0159] Besides, two integral areas of suitable size are prepared, and the restrictive number of children LC and the 
restrictive number of generations LG are set therein. 

[0160] When indicated with the above 3 bits, the settings of the modes are the any-copy = 000, one-copy = 001 . no- 
more-copy = 101 and never-copy =111. Besides, restricting only the number of children is set as "010 w , restricting only 
the number of generations is set as "001", and restricting both the numbers is set as "01 1" at the time of start 
[0161] Fig. 31 is a flowchart showing a process which is executed using the extended copy control modes, in compli- 
ance with a request for video recording. In the figure, control codes set for a child (a duplicate disk) are succeeded bv 
mark T. 

[0162] Referring to Fig. 31 , when the process has been started, the video recording request is accepted at step S50. 
At next step S51 , the value of the code CCCO indicating whether or not duplication or copying is permitted is decided. 
If the value is "1 ", the video recording is rejected at step S52, whereupon the process is ended. 
[0163] On condition that the duplication is permitted, the contents of 3 bits for a child are set to be the same as those 
for a parent, namely, for an original disk at step S53. Thereafter, the value of the code CCC1 indicating whether or not 
the number of copies is restricted is decided at step S54. If the code CCC1 is M V\ the number of copies is limited, and 
hence, a value with "1 " subtracted from the value LC for the parent is set as the copy restriction number LC# for the child 
at step S55. Subsequently, whether or not the number LC# has become "0" is decided at step S56. When the number 
LC# has become "0", the value of the code CCCO for the child is set at "1 " indicative of the barring of duplication at step 
S57. Then, the control flow shifts to the processing of step S58. The control flow also shifts to the processing of step 
S58 when the code CCC1 is not T at step S54, and when the number LC# is not "0" at step S56. 
[0164] At steps S58 ~ S61 , processing similar to that of steps S54 - S57 is executed as to the number of generations. 
A value with "1 - subtracted from the value LG for the parent is set as the generation restriction number LG# for the child. 
Lastly, at step S62, the video recording for the child or the duplication is executed, and copy control codes for the child 
are set. Then, the process is ended. Next, steps S63 - S68 checks if the parent cannot form a child (copy) any longer, 
in such a way that the copy restriction number LC for the parent is decreased M 1 M in succession after the end of the video 
recording. Copy control codes for the parent are set at step S68. 

[0165] Incidentally, it is also allowed to employ a contrivance in which "1 " is not subtracted from the number LC at step 
S55. Further, it is allowed that the restriction of the number of children to be set at the time of the creation of the parent 
by.the copyright holder be stored as a value "LC0" beforehand, and that the value H LC0 H be substituted into the number 
LC# at step S55. On this occasion, steps S56 and S57 are dispensed with, and step S55 is followed by step S58. 
[0166] Fig. 32 is a diagram for explaining the course of the creation of an original moving image in a package medium 
by a creator, the circulation or distribution of the package medium, and video reproduction/recording of the moving 
image by a viewer, the package medium being an MO disk, a DVD or a digital video tape by way of example. Fig. 32 
consists chiefly in altering the illustrative form of Fig. 27, and it is therefore omitted from detailed description. In the case 
of Fig. 32, before the moving image is stored in the package medium, it is subjected to data compression based on the 
MPEG scheme at step S73. In reproducing and displaying the moving image, the compressed data are expanded by 
the MPEG scheme at step S77. In contrast, in case of a request for recording and duplication, the compressed data are 
duplicated without the processing of the MPEG scheme. It is as thus far stated that the copy control code is checked at 
step S81 prior to the recording and duplication, whereupon the recording is sometimes rejected. Besides, a package 
medium obtained by the recording and duplication in this way is delivered to a friend or the like and is distributed among 
consumers. 

[0167] Lastly, the architecture of a computer system which incarnates a system for inserting a watermark and copy 
restriction information according to the present invention, and how to load programs in such a computer system from 
storage media, the programs serving to insert the watermark and the copy restriction information, will be described with 
reference to Fig. 33 and Fig. 34, respectively. 

[0168] Fig. 33 is a block diagram showing the architecture of the computer system which incarnates the system for 
inserting the visible watermark and the copy restriction information. Referring to the figure, the computer system 
includes a central processing unit (CPU) 91 which controls the whole system, a memory 92 which stores therein a mov- 
ing image bearing the inserted watermark and copy restriction information, and control information items for selecting 
the numbers of watermark insertion frames and for determining the insertion positions of the watermark information, a 
main storage 93 in which the programs for inserting the watermark information and the copy restriction information are 
temporarily stored in order to be run by the CPU 91 , an input/output device 94 which serves to input and output moving 
image data, etc., and a communication control device 95 which serves to transmit and receive the moving image data, 
etc. inputted and outputted through, for example, a network. 

[0169] Fig. 34 is a diagram for explaining the loading of a computer with the programs for inserting the watermark 
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101 and a memory 102. Hie compute prop* 101 car „ rough a „ eW ort> 103. 

or a hard disk by way of example. „ WrMrU in^rmation and the cooy restriction information, a program 

[0171] Besides the progr a ms for jnser*ng ^ ete are stored 

for executing the ^S^^S^ can aiso be performed by ioading the programs 

in the removable storage medium 104. me w ^ er ™ m 104 is anv storage medium which is commercially 

into the computer 100. Usable as the removable «W ™*w™W4 • ^ (co mpact-disk read only mem- 

avai.ab.e and which is ^^^^^ tSST* wLrmaiTnfor'mation. etc. can be per- 
ory), an optical disk or a magneto-optical dl ^ J"7j w ^J"^ j nform ation and the copy restriction information are 
formed in such a way that the programs fonnsert ng J^Z^^S^ network 1 03 so as to be loaded, 
sent from the side of a program provider to ^.^^*^2^ew8dSne for protecting the copyright of a moving 
[0172] As described above, » e a very wide range utilizing mov- 

cretely, the following effects are mentioned: 

(a.^eaoh.me.oimecw^pio^ 

.^,Sn^^ 

to social adjustments. ™ 0 * ort ori riata area and are handled in hardware in a read- 

W copy -*1?^25££SS2 CSS £n Am «• • — . „ 

out/reception device. These are effectives proiew u transmission of broadcast/communication or the 

iua9 ad in a short time <'^^*i^^'^on "Sas ooneemional eohemes ae th. any-oopy mode, 
broadcast/communication schemes, etc. adopt.no the new scheme of_ toe present invent o 

^maionsuohaameoc^W^ 
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Moreover, the watermark is less prone to be deteriorated by various image processing, and the copyright indication 
can be sustained for long. 

Claims 

1. A watermark insertion device for inserting watermark information into a moving image composed of a plurality of 
frames, comprising: 

frame selection means (1) for selecting frames into which the watermark information is to be inserted, at irreg- 
ular intervals from among the plurality of frames; and 

watermark-information insertion means (2) for inserting said watermark information into the selected frames. 

2. The watermark insertion device according to claim 1 , wherein 

said watermark information is visually perceivable when the moving image is brought to a standstill, and said 
frame selection means (1 ) selects the frames so that the inserted watermark information may not be recogniz- 
able in said moving image. 

3. The watermark insertion device according to claim 1 or 2, wherein 

said frame selection means (1) determines the frames to-be-selected by use of random number values which 
are generated under a controlled condition. 

4. The watermark insertion device according to any preceding claim, further comprising: 

moving-image storage means for storing therein the moving image in which said watermark information has 
been inserted, together with information indicative of the frames in which said watermark information has been 
inserted. 

5. The watermark insertion device according to any of claims 1 to 3, further comprising: 

moving-image storage means for storing therein information indicative of the controlled condition, together with 
the moving image in which said watermark information has been inserted. 

6. The watermark insertion device according to any preceding claim, wherein 

said watermark-information insertion means (2) determines those positions in said selected frames at which 
said watermark information is to be inserted, by use of random number values which are generated under a 
controlled condition. 

7. The watermark insertion device according to any preceding claim, wherein 

said watermark-information insertion means (2) selects any of a plurality of sorts of watermark information pre- 
pared beforehand and then inserts the selected watermark information. 

8. The watermark insertion device according to any preceding claim, wherein 

said watermark-information insertion means (2) selects any of a plurality of sorts of patterns prepared before- 
hand and then inserts the selected pattern, in addition to said watermark information. 

9. The watermark insertion device according to any preceding claim, wherein 

said watermark-information insertion means (2) inserts patterns which correspond to numbers of the frames 
selected by said frame selection means (1), the numbers indicating positions of said selected frames in the 
moving image, in addition to said watermark information. 

10. The watermark insertion device according to any preceding claim, wherein 
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said frame selection means (1) se.ects the frames so that said selected frames may not be successive. 

11 . The watermark insertion device according to any preceding claim, wherein 

said watermark-information insertion means (2) inserts said watermark information into at least one frame 
w^hTsISessive to any of the frames selected by said frame se.ect.on means (1). 

12. The watermark insertion device acceding to any preceding claim, wherein 

said watermark-information -ertion ^ 0) ^^^"2^^^^ ^25 

15 13 The watermark insertion device acceding to any preceding claim, further comprising: 
20 14 The watermark insertion device according to any preceding claim, further comprising: 

25 1 5. The watermark insertion device according to claim 1 4. wherein 

the copy restriction information is made detectable in hardware independently of satf watermark information. 
16 The watermark insertion device according to claim 14. wherein 

SSthe new copying, and a mode which rejects said copying entirely. 

18. The watermark insertion device according to any preceding claim, wherein 
said frame selection means (1) se.ects a first prescribed number of frames every second prescribed number 
of frames. 

19. The watermark insertion device according to any of claims 1 to 17. wherein 
said frame selection means (1) selects the frames in the number of random number values generated under a 
controlled condition, every second prescribed number of frames. 

20. A moving-image recording apparatus for recording a moving image composed of a p.uralrty of frames, comprising: 
judgement means for judging if the mo^^ 

copy means for copying said moving image on the basis of the judgement of said judgement means. 

21. A method of inserting watermark information into a moving image composed of a plurality of frames, comprising 
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selecting frames into which the watermark information is to be inserted, at irregular intervals from among the 
plurality of frames; and 

inserting said watermark information into the selected frames. 

5 22. The watermark insertion method according to claim 21 , wherein said watermark information has a size which is 
large enough to visually perceive said watermark information when the moving image has been brought to a stand- 
still. 

23. The watermark insertion method according to claim 21 or 22, wherein the insertion of said watermark information 
io into the frames is done in a quantity in which said insertion of said watermark information is not recognizable in the 

moving image. 

24. The watermark insertion method according to any of claims 21 to 23, wherein ninformation which restricts copying 
of the moving image is recorded together with said moving image in which said watermark information has been 

15 inserted. 

25. A computer-readable storage medium used to direct a computer inserting watermark information into a moving 
image composed of a plurality of frames, to perform the functions of: 

20 selecting frames into which the watermark information is to be inserted, at irregular intervals from among the 

plurality of frames; and 

inserting said watermark information into the selected frames. 



25 



26. A storage medium storing a moving image composed of a plurality of frames, wherein 

a moving-image storage medium storing the moving image in which visible watermark information being visu- 
ally perceivable when said moving image has been brought to a standstill is inserted in the frames at irregular 
intervals so that the watermark information may be visually unperceivable in said moving image. 

30 27. The moving-image storage medium according to daim 26, wherein copy restriction information indicative of a 
restrictive condition for copying of said moving image is inserted in that area of said moving image in which control 
data is recorded. 
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SY [0] ; AND 
FILE FD FOR SAVING THOSE PARTS OF ORIGINAL IMAGE 
WHICH HAVE BEEN HIDDEN BY WATERMARK) 



/ 



S30 



CREATE LIST OF WATERMARK INSERTION POSITIONS BY USE OF WATER- 
MARK CONTROL INFORMATION OF ORIGINAL COPYRIGHT HOLDER. AND 
STORE RESULTING LIST 
for k:-1 to N/n do I 

Ik Ck] :=floor(RANU(S Ck-1] )*n)+n(k-1)+1 

S Ck] -NEXT SEED VALUE GENERATED BY RANU 
Xk Ck] :=f loor(RANU(SX Ck-1] )*(Mx-Lx))+1 

SX Ck] :=NEXT SEED VALUE GENERATED BY RANU 
Yk Ck] :=floor(RANU(SY Ck-1] >*(My-Ly>)+l 

SY Ck] :=NEXT SEED VALUE GENERATED BY RANU 



S3! 



ACCESS MOVING IMAGE FILE FB OF DEALER B SUSPECTED OF PLAGIARY. 
AND FIND STARTING FRAME IB OF SUSPICIOUS PLAGIARIZED PART IN 
MOVING IMAGE B. 
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FIND FRAME IA IN MOVING IMAGE A THAT CORRESPONDS TO ABOVE PART, 
AND CONFIRM CORRESPONDING RELATIONSHIP BETWEEN FRAME IB AND 
FRAM IA. 
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OBTAIN FIRST WATERMARK INSERTION POSITION Ik [k] BEYOND FRAME 
NO "IA" AND COMPARE WATERMARKED IMAGE A [Ik [k] ] AND COR- 
RESPONDENT IMAGE B ClBk tkl 3 IN MOVING IMAGE A AND SUS- 
PICIOUS MOVING IMAGE B WHILE BEING SAVED. 
(IF NECESSARY, THE (SAVED AND RETAINED) ORIGINAL IMAGE FD [k] 
BEFORE THE WATERMARKING IS ALSO UTILIZED. 
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JUDGE RESULT OF COMPARISON BETWEEN IMAGE A AND IMAGE B FROM 
AMONG FOLLOWING. AND RECORD JUDGEMENT: 

((a) THE WATERMARK OF THE COPYRIGHT A REMAINS IN THE IMAGE B. 

(b) THE MOVING IMAGE B HAS SKIPPED THE FRAME Ik [k] OF THE 
MOVING IMAGE A. 

(c) IN THE IMAGE B, THE COPYRIGHT INDICATION OF THE DEALER B 
IS WATERMARKED SO AS TO COVER WATERMARK PART OF THE IMAGE A. 

(d) THE IMAGE B DOES NOT CONTAIN THE WATERMARK OF THE IMAGE A. 
AND THE CORRESPONDENT PART IS CLOSE TO THE ORIGINAL IMAGE A 
BEFORE THE INSERTION OF THE WATERMARK. BUT IT IS UNNATURAL. 

(e) THE IMAGE B MATCHES COMPLETELY WITH THE ORIGINAL IMAGE A 
BEFORE THE INSERTION OF THE WATERMARK) 



J 
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JUDGE ABOVE RESULTS OF COMPARISONS SYNTHETICALLY. 
((1) THE CASE(a) EXISTS EVEN IF ONCE. 

- THE SUSPICION IS CONFIRMED. 

(2) THE JUDGEMENT (b) OR (c) IS MADE IN ALL THE COMPARISONS. 

- THE SUSPICION IS ALMOST CONFIRMED. 

(3) THE JUDGMENT (b) OR (c) IS MADE IN MOST OF THE COMPARI- 
SONS. AND THE JUOGEMENT (d) OR SUSPICION IS VERY CERTAIN 

—THE SUSPICION IS VERY CERTAIN. 

(4) SOME OF THE COMPARISONS FALL UNDER THE CASE(b). (c) OR(d). 

- THE SUSPICION IS STRONG. 

(5) ALL THE COMPARISONS FALL UNDER THE CASE(e). 

- THE DEALER B WOULO HAVE OBTAINED THE MOVING IMAGE BEFORE 
BEING WATERMARKED BY THE ORIGINAL COPYRIGHT HOLDER A. 
OR WOULD HAVE REMOVED THE WATERMARK BY ANY METHOD. ) 
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